[FLINK-40089][table] Implement JSON_LENGTH function by VasShabu · Pull Request #28688 · apache/flink

VasShabu · 2026-07-08T12:33:59Z

What is the purpose of the change

This pull request adds support for the built-in SQL function JSON_LENGTH, which returns the number of top-level elements in a JSON array or object. Scalars have a length of 1, and NULL input returns NULL. Nested arrays or objects are not counted recursively.

Brief change log

Added JsonLengthFunction implementing the JSON_LENGTH built-in function
Registered JSON_LENGTH as a built-in function in the function catalog
Added test cases for JSON_LENGTH in JsonFunctionsITCase

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

mvn test -pl flink-table/flink-table-runtime -Dtest=JsonFunctionsITCase
Added test cases for JSON_LENGTH covering scalars, arrays, objects, and NULL input in JsonFunctionsITCase

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs (sql_functions.yml)

Was generative AI tooling used to co-author this PR?

Yes (Claude)

flinkbot · 2026-07-08T12:43:54Z

CI report:

9f85f67 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

raminqaf

Thanks for the contribution @VasShabu! Left some comments!

raminqaf · 2026-07-08T12:47:39Z

+    public static Integer jsonLength(String input, String pathSpec) {
+        return jsonLength(jsonApiCommonSyntax(input, pathSpec));
+    }
+
+    private static Integer jsonLength(JsonPathContext context) {
+        if (context.hasException()) {
+            return null;
+        }
+        final Object value = context.obj;
+        if (value == null) {
+            return null;
+        }
+        if (value instanceof Map) {
+            return ((Map<?, ?>) value).size();
+        }
+        if (value instanceof Collection) {
+            return ((Collection<?>) value).size();
+        }
+        return 1;
+    }


Why to we need this? Don't we have already the runtime class JsonLengthFunction?

raminqaf · 2026-07-08T12:48:56Z

+        super(BuiltInFunctionDefinitions.JSON_LENGTH, context);
+    }
+
+    public @Nullable Integer eval(@Nullable StringData jsonInput) {


nit: here and everywhere else make parameters/variables that are immutable final (not enforced on Flink but nicer to read).

Suggested change

public @Nullable Integer eval(@Nullable StringData jsonInput) {

public @Nullable Integer eval(final @Nullable StringData jsonInput) {

gustavodemorais · 2026-07-08T13:42:48Z

        }
    }

+    public static Integer jsonLength(String input, String pathSpec) {


We have these utilities here and some other in JsonLengthFuntion.java itself. Check if we don't already have utilities for performing the length check. If not, check if it makes sense to move the new private utilities you added to this file

gustavodemorais

Thanks for the contribution, @VasShabu! Good job on the first version of the PR. We'll go through some iterations to make sure things look good

Apart from what we already flagged above: we have two implementations of the same function. eval(json) uses a new Jackson readTree, eval(json, path) uses the existing Calcite path in SqlJsonUtils. Different parsers + different data models = they can disagree on the same input. We can make the design more consistent by using only one path. Duplicate ObjectMapper - reuse the configured one in SqlJsonUtils instead of a second bare instance. This is important because built-in functions are on the hot path. The logic might be invoked millions of times in a couple of seconds and we have to make sure they allocate as little as possible and their performance are optimized or else the engine will be wasting resources.

That said, In this case, we could go two paths

Best performance: We build a custom parser to optimize it for length checking by skipping the children. MAPPER.getFactory().createParser(str), read first token; if START_ARRAY/START_OBJECT walk top level counting, skipping children via parser.skipChildren(). No tree alloc. Best for length-only.
Simplest reuse the shared SqlJsonUtils path so both overloads share one parser.

Take a look at both paths and let me know what you think

gustavodemorais · 2026-07-08T14:28:37Z

+        } else if (jsonNode.isTextual()
+                || jsonNode.isNumber()
+                || jsonNode.isBoolean()
+                || jsonNode.isBinary()) {


Isn't this check dead code? Can a jsonNode be binary? You can try to write a test to check if it's dead code or it's possible

snuyanzin

Are we sure the selected approach is the right one?
With this approach the same JSON will be parsed several times for queries like

SELECT JSON_LENGTH(my_json), JSON_LENGTH(my_json), JSON_LENGTH(my_json) FROM...

or

SELECT JSON_LENGTH(my_json), JSON_VALUE(my_json, '$.a') FROM...

snuyanzin · 2026-07-08T18:21:32Z

+    table: jsonLength(jsonObject[, path])
+    description: |
+      Returns the number of elements in a JSON document, or the length of the value at the specified path if one is provided. 
+      Returns NULL if the argument is NULL or the path does not locate a value.


this does not look true
simple tests show different behavior

SELECT json_length('$'); SELECT json_length('{');

each returns -1 and the doc says nothing about this or did I miss anything?

snuyanzin

More findings:

Doc doesn't reflect the real behavior: it returns -1 in case of invalid json, while docs says about NULL, e.g. query SELECT JSON_LENGTH('null');
Strange handling of null values
like a query
```
SELECT json_length('{"a":[true, false, null]}', '$.a[1]')
```
both in MySQL and in Flink it returns 1
now
```
SELECT json_length('{"a":[true, false, null]}', '$.a[2]')
```
in MySQL it returns 1, in Flink it returns -1 why?

Seems need to check what the behavior should be in case of lax/strict since right now it is weird

SELECT json_length('{"x": {"a":1}, "y": {"a":[2, 3]}}', 'lax $.*.a');  -- returns 2

SELECT json_length('{"x": {"a":[1, 2, 3, 4]}, "y": {"a":[2, 3]}}', 'lax $.*.a');  -- returns 2

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a'); -- returns 3

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[0]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[1]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[2]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[3]');  --returns 0

davidradl · 2026-07-13T13:57:17Z

        """
        return _unary_op("jsonUnquote")(self)

+    def json_length(self, path = None) -> 'Expression':


It seems that many databases throw an error, I am curious why we are not throwing an error. Also I see that there are best practises that we should document like using IF(string_column IS JSON, to protect against non json and avoid potential errors.

…single parse

snuyanzin · 2026-07-14T09:26:45Z

+        final JsonPathContext context = jsonApiCommonSyntax(parsedInput, pathSpec);
+        final Object value = context.hasException() ? null : context.obj;
+
+        if (value instanceof LinkedList) {


is there a reason we expect LinkedList instead of just List?

snuyanzin · 2026-07-14T09:27:29Z

+ * same input expression the parsed JSON context is reused via a shared member variable, so a given
+ * JSON string is parsed at most once per row regardless of how many JSON functions read it.
+ */
+class JsonLengthCallGen extends CallGenerator {


ideally we should not have new classes in scala

snuyanzin · 2026-07-14T09:28:13Z

+logger.codelog.name = org.apache.flink.table.runtime.generated.CompileUtils
+logger.codelog.level = DEBUG
+logger.codelog.appenderRef.test.ref = TestLogger


it might be helpful for local debug, however why do we need in repo and making it executable for every ci run?

snuyanzin · 2026-07-14T09:29:49Z

+                        jsonValue, // f0: existing resource JSON
+                        "{\"a\":1,\"b\":2}", // f1: object
+                        "[1,2,3]", // f2: array
+                        "\"abc\"", // f3: scalar string
+                        "null", // f4: JSON null
+                        "{", // f5: invalid JSON
+                        ((String) null), // f6: SQL NULL
+                        "$",
+                        "{\"a\":[true, false, null]}", // f8
+                        "{}", // f9: empty object
+                        "[]") // f10: empty array


not sure we need all these comments, what a value it could give to the reader who sees that {} is an empty object and then a comment telling same?

snuyanzin · 2026-07-14T09:30:26Z

+        assertThat(rows).containsExactlyInAnyOrder(Row.of(4, "account"), Row.of(4, "admin"));
+        assertThat(countJsonParse(extractGeneratedCode(sql)))
+                .as("JSON_LENGTH + JSON_VALUE on the same input should parse once")
+                .isEqualTo(1);


Suggested change

.isEqualTo(1);

.isOne();

nit

raminqaf

Another pass

raminqaf · 2026-07-14T09:59:01Z

+            return null;
+        }
+        final JsonPathContext context = jsonApiCommonSyntax(parsedInput, pathSpec);
+        final Object value = context.hasException() ? null : context.obj;


if context.hasException() is true, then value would be null. We should probably then short circuit and return null instead of continuing and risking a NPE

raminqaf · 2026-07-14T09:59:43Z

+        if (context.hasException() || value == null) {
+            return pathExists(parsedInput.obj, pathSpec) ? 1 : null;
+        }


See comment: https://github.com/apache/flink/pull/28688/changes#r3577941235 and move it higher up

raminqaf · 2026-07-14T10:01:11Z

+        return jsonLengthValue(value);
+    }
+
+    private static Integer jsonLengthValue(final Object value) {


Here we never return null so we can use primitiv int

Suggested change

private static Integer jsonLengthValue(final Object value) {

private static int jsonLengthValue(final Object value) {

raminqaf · 2026-07-14T10:04:23Z

+     * whether the matched value is a JSON null literal. Uses {@link Option#AS_PATH_LIST} so the
+     * result is the list of matched canonical paths, which is empty iff the path does not exist.
+     */
+    private static boolean pathExists(Object json, String pathSpec) {


nit: Since you are at it

Suggested change

private static boolean pathExists(Object json, String pathSpec) {

private static boolean pathExists(final Object json, final String pathSpec) {

raminqaf · 2026-07-14T10:08:01Z

+            if (matched.size() != 1) {
+                if (context.mode == PathMode.STRICT || matched.isEmpty()) {
+                    return null;
+                }
+                return matched.size();
+            }


Could you please elaborate on this? In which conditions we get a null and in which conditions we get the size of the matched?

[FLINK-40089][table] Implement JSON_LENGTH function

200b15a

raminqaf reviewed Jul 8, 2026

View reviewed changes

gustavodemorais reviewed Jul 8, 2026

View reviewed changes

snuyanzin requested changes Jul 8, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 8, 2026

snuyanzin reviewed Jul 8, 2026

View reviewed changes

snuyanzin requested changes Jul 8, 2026

View reviewed changes

davidradl reviewed Jul 13, 2026

View reviewed changes

[FLINK-40089][table] Fix JSON_LENGTH null handling, used codegen for …

9f85f67

…single parse

snuyanzin reviewed Jul 14, 2026

View reviewed changes

Comment thread ...k-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java

snuyanzin reviewed Jul 14, 2026

View reviewed changes

Comment thread ...k-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java

snuyanzin reviewed Jul 14, 2026

View reviewed changes

Comment thread ...flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/SqlJsonUtils.java

raminqaf reviewed Jul 14, 2026

View reviewed changes

	public @Nullable Integer eval(@Nullable StringData jsonInput) {
	public @Nullable Integer eval(final @Nullable StringData jsonInput) {

	private static Integer jsonLengthValue(final Object value) {
	private static int jsonLengthValue(final Object value) {

	private static boolean pathExists(Object json, String pathSpec) {
	private static boolean pathExists(final Object json, final String pathSpec) {

Uh oh!

Conversation

VasShabu commented Jul 8, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gustavodemorais left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

flinkbot commented Jul 8, 2026 •

edited

Loading

snuyanzin left a comment •

edited

Loading

snuyanzin Jul 8, 2026 •

edited

Loading

snuyanzin left a comment •

edited

Loading